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Introduction 

Under the larger scope of the National Partnership for 
Quality Afterschool Learning, SEDL funded three awardees 
to carry out large-scale randomized controlled trials (RCT) 
assessing the efficacy of promising literacy curricula in 
afterschool settings on student academic achievement. 

SEDL provided analytic and technical support to the RCT 
studies through its Afterschool Research Consortium (ARC), 
composed of SEDL researchers, key staff from each research 
project, and experts in the field 1 . The ARC convened at least 
twice a year to discuss accomplishments, challenges, and 
solutions to the implementation of the funded RCT study 
designs. The ARC has been committed to the dissemination 
of information to the field about the effective use of 
rigorous experimental research approaches in applied 
afterschool settings. This research brief is the second in 
a series of papers intended to address some of the key 
challenges faced by awardees in order to provide insights to 
the research and practice community. The information for 
this brief includes lessons extracted from ARC discussions 
and activities, guidance provided by ARC experts, site 
visits and interviews conducted by SEDL, and the findings 
described in the larger literature of afterschool research. 

This brief is organized according to the primary challenges 
undertaken by the ARC during the early funding and 
implementation period, which involved two interrelated 
topics: difficulties with the recruitment of sites and 
challenges with implementation of curricula that had been 
adapted to fit the afterschool setting. These issues are part 



of a larger set of often-uncovered mechanisms or "black box" 
events occurring during implementation that can insidiously 
contribute to dampened treatment effects in these applied 
studies. This discussion is aimed at contributing practical 
information about conducting RCT studies in applied 
settings, suggesting strategies that might help circumvent 
a few of the multiple ways in which study effects are 
threatened, and broadening such discussions in the field. 

Brief Overview of the Key Issues Related to 
Recruitment and Implementation 
The use of RCT designs to evaluate the impact of academic 
curricula in applied afterschool settings is a relatively new 
and demanding approach in program evaluation research. 

As more RCT studies are undertaken, the little known 
practical aspects of conducting this type of research are 
emerging; the challenges that must be overcome, the 
solutions to common problems associated with RCT studies, 
and the specific elements of afterschool settings that 
influence research designs. We begin by briefly describing 
study elements that are salient for this research brief on 
recruitment and implementation issues and strategies that 
emerged over the two-year course of the studies. 2 

In the spring of 2006, SEDL issued a request for proposals 
for the rigorous evaluation of the efficacy and impact of 
promising afterschool interventions using RCT methodology. 
Three research projects were funded to test promising 
reading interventions and their impact on student 
achievement in contrast to the programs or practices that 



1 Fred Doolittle, MDRC; Elizabeth Reisner, Policy Studies Associates, Inc.; and Peter Witt, Texas A & M University participated in the ARC as Technical Working 
Group members. 

2 Afterschool Research Brief #1 provides detailed information about the study criteria, the selection process, and a discussion of the state of the field of 
afterschool research using RCT designs. 
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were operating normally in the afterschool programs. 
All three projects were testing reading interventions, 
but the early recruitment and implementation phases 
for each study were influenced by "behind the scenes" 
issues that were tied to unique and fundamental 
elements of the individual study designs. 

For example, in the study conducted by the Success for 
All (SFA) Foundation, the Adventure Island curriculum, 
an afterschool reading program based on the SFA 
reading model (Slavin & Madden, 2001) with a focus 
on the components that are identified as common 
deficiencies among struggling readers, was proposed 
with a student population of English language learners 
(ELLs) in grades 2-5 3 . The inclusion of ELL students 
in the study design raised insurmountable obstacles 
during first-year attempts at recruitment of afterschool 
sites. During the extended recruitment period, a 
refinement of strategies was developed based on 
each attempt and the sample was drawn successfully 
from four majority-Flispanic and one majority-African- 
American schools in Texas. This challenge transformed 
the unexpected difficulties into a platform for 
developing a set of recruitment strategies for targeting 
special populations. The strategies used to overcome 
these challenges will be discussed in greater detail in 
the recruitment section. 

In the second study designed by The Center for 
Evaluation and Education Policy (CEEP) at Indiana 
University, the impact of previously established 21st 
Century Community Learning Center (21st CCLC) 
programs across Kentucky were to be compared with 
similar 21st CCLC programs that included the Voyager 
Passport Reading program (Passport) in grades 2-5. 
This study's early challenges with site recruitment were 
hinged on the design element of recruiting programs 
across an entire state. Although a sufficient number 
of counties across the state had agreed to participate 
initially, at the start of the school year afterschool 
programs encountered unexpected insufficient staffing 
resources, scheduling problems, and related difficulties 
with implementing the RCT design. The attempt to 
recruit sites across an entire state brought about a 
unique set of issues; while most of the issues that 
interfered with successful recruitment were related 
to insufficient resources, many of them included 
challenges related to geographic characteristics such 



as remoteness of rural settings. A fuller discussion of 
these challenges is presented in later sections. 

The third study, undertaken by MPR Associates, 

Inc., was designed to compare Scholastic's READ 
180 program with standard afterschool services in 
Brockton (MA) public schools. Prior to SEDL funding, 
the William T. Grant Foundation had funded the first 
year of a study to examine the impact of READ 180 in 
afterschool classrooms in three schools in Brockton, 
which revealed positive effects on oral reading fluency 
for some students in the treatment group. SEDL 
funded a replication study using four "new" schools 
in the same district that had never used the READ 
180 program. The study drew upon experience in the 
previous year to recruit approximately 300 students in 
grades 4, 5, and 6. The MPR study also benefited from 
its established connections with the developer, the 
district, and its "pilot" year findings to successfully 
overcome a number of typical issues that occurred 
during the SEDL study. 

The specific issues that influenced recruitment were 
the leading indicators of how well the implementation 
phase would proceed. The studies' start-up activities 
were influenced by whether key personnel in the 
afterschool sites had accurately assessed and were 
able to meet the conditions needed to implement the 
interventions. In several cases there were instrumental 
advocates for the projects that went above and 
beyond what is typically required by their roles to 
find resources, talk with school administrators, and 
personally communicate with study staff about the 
progress of the studies. During the two-year funding 
period, the ARC provided a collaborative setting in 
which to troubleshoot a number of substantive issues, 
and it is from those discussions, reports provided 
by the project staff, and site visits and interviews 
conducted by SEDL that the information in this brief 
was obtained. 

The ARC produced this series of briefs as an outlet 
to document the common issues that affected the 
success of rigorous evaluations in applied settings and 
to share strategies that worked in particular settings 
and conditions. All of the issues discussed in this brief 
are specific to large-scale efficacy trials in afterschool 
programs, however most of them generalize to issues 
that arise commonly in conducting RCT studies in 



3 At the start-up of SEDL funding, SFA's Adventure Island was being evaluated in an afterschool study conducted by MDRC, without this sample 
specification. 
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other areas. Along with each set of issues we outline 
a number of strategies untaken by these projects that 
were either successful or provide knowledge gained 
through attempts to overcome problems. 

Recruitment of Afterschool Sites into 
Large-Scale RCTs 

Key Issues 

The following issues arose during first-year attempts to 
recruit afterschool sites. Of the problems encountered 
across the projects, the three issues described below 
were the most common and notable. 

Geographic dispersion of potential sites. One of 

the defining features of the future success with 
recruitment was the geographic dispersion of potential 
sites. For example, in the summer and fall of 2006, 

SFA staff contacted district administrators to identify 
sites that would benefit from a study serving ELL 
students. After following up on leads in California, 

New York, Maryland, and Texas (Dallas and Fort 
Worth), issues that arose eliminated schools from this 
potential list. As it turned out, if more schools from 
the initial pool had signed letters, another challenge 
to final recruitment activities would have been the 
geographic complication that only 3 of the 14 schools 
in the initial pool were in the same district. The other 
schools were in separate cities, which would have 
required a substantial travel budget and significant 
coordinator time given the distance between sites and 
the related training, scheduling, and travel costs for 
trainers and testers. Travel to the sites at the time of 
recruitment may have improved chances that schools 
would have signed agreements, but travel to each site 
to explain the study and meet with critical support 
contacts would have been cost-prohibitive even before 
the study began. 

Knowledge of district policies governing special 
programs. District policies regulating special 
programs are especially influential in the planning 
for recruitment of special student populations. Given 
that SFA's target population was ELL students, their 
successful recruitment was influenced by district 
level policies governing ELL program types, such as 
bilingual and transitional. As is typical in recruitment 
protocol, research staff followed up with administrators 
and in some cases conducted district meetings for 
representatives from interested schools to ensure that 
they understood the conditions of the study. Two large 



districts in Texas agreed to participate during the 
summer of 2006, but later that fall, SFA received a 
letter withdrawing them from the study because district 
administrators of the bilingual education programs 
discovered that Adventure Island did not include any 
Spanish instruction, a key requirement of their bilingual 
education approach. Although district approval had 
been received, the bilingual education administrator 
had final authority and pulled the agreements. 

Afterschool program budget constraints. Another 
key issue that impacted study recruitment was limited 
school resources (e.g., transportation and staff). A 
large district in Maryland did not participate due to 
lack of funds for staffing and operating afterschool 
programming to provide Adventure Island for four 
days per week. Also, transportation would have 
been a particular financial and strategic challenge. 
Unanticipated resource shortfalls for afterschool 
programs impacted CEEP's study recruitment activities. 
At the end of the recruitment period, the overall 
sample size for the study was significantly lower than 
project staff had initially projected for a number of 
reasons. Overall, participation rates in the afterschool 
programs were significantly lower than anticipated, in 
some cases due to funding issues (e.g., insufficient 
funds to provide transportation). Given that Kentucky 
is characterized by high rurality, transportation costs 
weighed heavily in the sites' ability to operate their 
programs. Additionally, many schools were insufficiently 
staffed to accommodate the student/teacher ratio 
required by the afterschool intervention program. Eight 
counties withdrew from the study due to insufficient 
staff and also because of their inability to devote 45 
minutes to one hour of the afterschool program session 
to the Voyager Passport program. For studies of this 
kind, resources must be in place and remain consistent 
for the duration of the study, conditions that many 
under-funded afterschool programs cannot meet. 

Key Strategies 

Number and location of sites aligned to funding 
levels. The recruitment of afterschool program sites 
within close proximity of one another mitigated 
several demands on resources. In the SFA study, 
connections through professional networks identified 
an associate superintendent who oversaw special 
programs in her district and was an advocate of 
research. Aside from one large urban school site 
outside of this district, all program sites were under 
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the purview of this supportive district administrator, 
and within dose driving distance from one another. 
These two factors, administrator advocacy and the 
majority of sites within one district and in close 
proximity to one another, proved to be the critical 
strategy for recruitment success. MPR's program sites 
also were located within close proximity in one school 
district, and their first year study funded from a source 
other than SEDL had taken place in the same district. 
Relationships MPR built with the school district in the 
pilot year were drawn upon to recruit new schools for 
the SEDL study. These studies were efficacy field trials 
that typically require that sites be recruited from one 
to two districts rather than statewide, and further, the 
amount of funding for these studies was modest. The 
number and location of sites, when closely aligned to 
funding levels, improves the likelihood that sites can 
be recruited efficiently and effectively (i.e., reasonable 
investment of resources to explain the study in person, 
answer questions, and get initial "buy-in" from the 
administrators), and may increase the chance that 
district level administrators are invested in the overall 
success of the project. 

Use of "insider" contact information and support. 

Background research should be conducted, including 
the use of "insider" contact information and support, 
to identify special program intervention specifications 
and tie them to any district policy implications. 

SFA staff found an advocate for their study through 
contacts from colleagues to determine a target sample 
of districts that met study criteria, including district 
policy information that improved the fit between the 
intervention specifications and the sites' willingness 
to participate. Through this process, SFA successfully 
recruited schools with suitable afterschool programs 
in two large Texas districts as a result of joint efforts 
between SFA, SEDL, and the county-level administrator 
of special programs. For studies requiring special 
student samples that invoke district policies aimed 
directly at them, the first successful recruitment 
strategy included forming an allegiance with a district- 
level contact with significant administrator knowledge 
and power to assist with recruitment. In this case, 
one major district contact helped with the recruitment 
of the final sample of schools. This contact was also 
instrumental in the maintenance of the study sample 
during the implementation phase. This strategy worked 
for a small study, however, research on district policies 
that govern special elements of the intervention and 



study design should be undertaken before recruitment 
activities begin. 

Individual meetings with interested schools. 

Recruitment efforts should involve individual visits 
with each school to provide important background 
information about support requirements essential for 
or to improve program implementation. Feasibility 
visits would allow the evaluation researchers to build 
rapport, verify information about the core elements 
of the program, explain research requirements of 
the project (with a strong emphasis on the need to 
implement the program with fidelity) to school and 
program administrators, and to address administrators' 
concerns about the study design and resources to 
meet study specifications. Geographic proximity 
between the schools improved the financial and 
practical feasibility of these visits. Information about 
the timing of program implementation matched to 
afterschool program startup was critical to successful 
recruitment, and pertinent details regarding the overall 
operation of the programs was gained in individual 
meetings. The visits made during recruitment provided 
critical information about the types of training and 
supports that would facilitate implementation of the 
afterschool intervention, described more fully below. 

In general, creative solutions to unanticipated budget 
reductions were more likely to be successful when 
both the project teams and the district administrators 
had streamlined communication about the study and 
a sufficient amount of "buy-in" and rapport had been 
developed. Realistically, however, potential study sites 
that would struggle over the course of the study period 
to find appropriate resource levels needed for program 
implementation are better left off the recruitment list. 

Given the importance of successful site recruitment to 
the long-term success of RCT designs, budgeting for 
feasibility visits as well as adequate advance time to 
recruit sites is recommended and considered a high- 
priority investment in the study. Well-implemented 
studies begin with successful recruitment, which 
includes choosing sites that can attain and maintain 
the proper level of resources to implement the studies. 
In the end, successful recruitment of sites to conduct 
RCT studies is key to maintaining adequate samples 
to detect treatment effects by the end of the study. 
Further, enrolling the "best" pool of study participants, 
meaning those whose characteristics fit the intended 
population for the intervention (e.g., struggling readers, 
ELL students), will tie back to the program components 
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theoretically asserted to produce the outcome 
effects, which will ultimately address developers' 
concerns about whether the intervention received 
the best opportunity to produce intended effects on 
student outcomes. 

Early Implementation of Large-Scale 
RCT Studies in Afterschool Settings 

Key Issues 

The following issues were identified as key factors 
that influenced the implementation of the reading 
interventions. The discussion of implementation 
developed here is in general reference to the 
installation of the structured literacy programs 
in applied settings (Lipsey & Cordray, 2000). 
Implementation of the interventions was to be 
approached as a shared responsibility between 
evaluators and developers, with significant developer 
support considered part of the design for these 
efficacy trials. 

Degree to which key personnel were fully informed. 

Identifying and contacting all decision-makers who 
were empowered to finalize and uphold the agreement 
to participate in the study was a key strategy during 
recruitment. To the extent possible, and at every 
level of the sites' infrastructure, key personnel should 
be made aware of the requirements for successful 
program implementation and the overall study design. 
Key personnel include district administrators, school 
administrators, directors of afterschool programs, and 
all program staff that may have contact with study 
staff. In most cases, supportive site contacts were 
identified successfully, and those initial contacts were 
instrumental throughout the implementation phase. 

Even in cases where significant effort had been 
invested in informing key personnel about the study 
requirements some sites eventually withdrew, claiming 
that they had not understood the requirements of the 
study and were unable to participate. For example, 
after the early implementation phase and several 
weeks into the study, eight counties withdrew from 
the CEEP study after deciding it was too difficult to 
implement the intervention for reasons related to 
lack of teacher resources, insufficient program time 
to devote to the program requirements, and inability 
to meet the required ratio of teachers to students. 

As a result, site recruitment was staggered over a 
protracted period as new sites were approached for 



recruitment, delaying the professional training and 
implementation of the intervention. These issues were 
tied to a strained fit between sites' understanding 
of and ability to provide resources necessary for the 
implementation of the program. 

Familiarity with requirements and intent of highly 
structured research designs. Some administrators do 
not need much convincing that there is a high return 
on investment for participation in rigorous research 
projects, which is related to either previous positive 
experiences or their recognition that evidence-based 
decisions underpin their ability to improve academic 
achievement scores. The amount of time needed for 
program implementation, however, and the availability 
of afterschool staff to teach in the classrooms, are 
specific considerations that are typically the most 
resource intensive requirements that plague normal 
program operations. Afterschool programs on the whole 
have not been asked to implement rigorous studies of 
structured academic programs. However, changes to 
program materials must sometimes be requested and 
made to address problems with time allocations and 
staffing resource issues. 

For example, although the Adventure Island program 
was designed as 45-minute lessons, school visits at 
the beginning of the study led to the modification 
from four to three days per week given teaching 
staff availability and the need to use one model 
for all participating schools. Adventure Island was 
delivered three days per week for one hour each day 
so that the amount of instructional time met program 
requirements. The CEEP study staged staggered teacher 
trainings and continued to implement the program 
despite late start-up, which resulted in a low degree 
of on-model program implementation. All three 
studies made modifications to their program delivery 
to accommodate teacher and program requirements. 
None of the sites ran their afterschool program five 
days per week, which required some compression of 
program delivery. The MPR study, in particular, made 
modifications to their program delivery based on 
feedback from the afterschool program and teachers, 
moving from a four-day a week model to a compressed 
two-day a week model that still provided the same 
amount of time on the program. 

Relationship between evaluators and program 
developers. The quality of the relationship between 
evaluators and program developers became an 
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important issue over the first year. Significant developer 
support, required for these efficacy trials, enhanced 
the quality of program implementation when important 
information about the progress of the study was 
shared effectively and in a timely manner between 
the evaluation team and the program developers. A 
number of common problems with research in applied 
settings arose during the first year, and were confronted 
successfully when conversations between afterschool 
program staff, evaluation researchers, and developers 
were effective. One problem that arose for all three 
studies was the need for program adjustments to 
meet afterschool program constraints. Healthy, 
ongoing conversations between research stakeholders 
increased the likelihood of reaching consensus on 
program modifications or "tolerable adaptations" 
(Cordray, 2008). 

Key Strategies 

Fully informed key personnel on both evaluation 
research and program sides. Although it is common 
for unforeseen difficulties to develop, both the 
evaluation researchers and the program staff must 
have an accurate projection of the needs and 
requirements for high-quality program implementation. 
In the case of the SFA study, recruitment efforts 
had reached what seemed to be a good fit between 
sites and study requirements, when a last-minute 
budget shortfall brought into question whether 
enough afterschool program funding would be 
available to keep the program running throughout 
the school year. The district administrator who had 
been instrumental during recruitment was motivated 
to improve the quality of the district's after-school 
offerings, an objective shared by the project, which 
required additional financial resources to extend the 
afterschool program. This extension allowed for the 
implementation of the intervention program across 
the school year (i.e., the afterschool program was 
not funded originally to provide services through the 
end of the school year). In this case, the quality of 
communication between the study staff, the district 
administrator, and program sites improved the 
likelihood that the unforeseen problems would be 
resolved successfully. 

Loss of recruited sites may have been be pre-empted 
through presentations to several layers of stakeholders, 
preferably in person, to explain the design of the 
research, the basic elements of the program to be 



implemented, and provide a period of questions and 
answers. In the ideal, unanticipated resource shortfalls 
and budget constraints for afterschool programs may 
have a better chance of being solved creatively when 
the project advocates have agreed to participate, are 
fully informed, and want the study to be a success. 
Given the loss of significant portions of the first year 
cohort, implementation difficulties, and insufficient 
program resources to provide the needed teachers and 
program time for the intervention, CEEP and Voyager 
felt there were no alternatives other than to conclude 
the study. By the end of the first year, CEEP, Voyager, 
and SEDL all agreed it was not feasible to go forward 
with the original study as designed given that none of 
the program sites indicated that they could assure the 
resources necessary to participate in the study. 

Adaptability in response to rigorous research 
design and structured programming. The SFA and 

MPR studies were able to adapt to the afterschool 
conditions that affected program implementation 
without threatening the integrity of their research 
designs. The amount of time needed for the structured 
academic programs, teacher and administrator feedback 
on the program's fit to the afterschool setting, and the 
availability of afterschool staff to teach in classrooms 
led to adjustments in the delivery of the intervention 
programs. None of the sites ran their afterschool 
programs 5 days per week. In the SFA study sites, most 
programs had certified staff teaching afterschool three 
days a week and paraprofessionals on the fourth day to 
provide an additional day of afterschool programming. 
The three-day version of the Adventure Island 
curriculum was a slight modification, and therefore an 
acceptable adaptation to the program design, resulting 
in a need for adjustments to pacing of the lessons. 
Although the Adventure Island program called for 
45-minute lesson blocks, school visits at the beginning 
of the study led to the modification from four to three 
days per week given teaching staff availability and the 
need to use one model for all participating schools. 
Adventure Island was delivered three days per week for 
one hour each day, which met on-model instructional 
time requirements. 

Between the previous year of research funded by 
the William T. Grant Foundation and the first year of 
research under SEDL funding, Scholastic developed a 
new version of READ 180 (Enterprise) that incorporated 
changes in materials, emphasis, and technology. The 
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earlier version of READ 180 was more adaptable to 
a 60-minute period that fit well with the existing 
afterschool schedule; the new Enterprise version was 
more difficult to modify. Suitable modifications to the 
new program were found, while still maintaining the 
rigor of the research design. Changes from the first- 
year four-day per week/60 minutes per day model 
were made in the second year to include a two-day 
per week/90 minute model (except for one school 
that offered a four day per week/90-minute model). 

The two day per week model was well-received by the 
afterschool teachers, and was expected to increase 
attendance rates and decrease attrition rates, while 
permitting greater flexibility in scheduling the daily 
routine of the afterschool program. 

Successful communication between evaluators, 
afterschool program staff, and developers. The 

conditions of these efficacy trials required that the 
RCT awardees work with the program developers and 
the afterschool programs to implement a promising 
intervention program under ideal conditions. 

Afterschool settings are characterized by a number 
of conditions that add complexity to the delivery of 
structured academic programs. Miscommunication or 
spotty information between evaluation researchers, 
afterschool staff, and developers raises anxiety that 
the project is either not going well or that the program 
is not being implemented well enough to give it a 
reasonable chance to show positive effects. Again, 
an upfront investment of time and energy to develop 
strong alliances in the effort to achieve successful 
program implementation pays off in the end. 

As in any rigorous research design applied to school 
settings, a number of issues arose immediately that 
never were resolved and negatively impacted the 
CEEP study. Sample recruitment problems, rolling 
enrollment, and staggered teacher training and 
implementation of the intervention were the most 
critical threats to successful implementation of the 
Voyager program. . More generally, the degree to which 
rapport and trust can be built between the key staff 
on the evaluation and developer teams, in addition 
to the afterschool staff, the greater the chance that 
reasonable solutions can be found for the problems 
likely to occur in all applied settings. 

Additional strategies for successful 
implementation. There was a cadre of "unsung 
heroes" working behind the scenes in the successful 



projects. The local district coordinator overseeing the 
SFA program sites was a dedicated point person with 
deep knowledge of district operations and history, 
its administrators, and its lead teachers. She and the 
professional development trainer provided on-site 
technical assistance. 

Other staff (research contacts at each program site, 
program administrators) involved in both the SFA and 
MPR studies were willing to devote time and energy 
to facilitate success with the rigorous implementation 
requirements, including tracking down students 
who were absent from the classroom, talking with 
parents, and providing candid feedback to study 
staff that assisted the development of modifications 
that improved program implementation. Research 
contacts, site leaders who were liaisons with the 
SFA district coordinator, were trained and served as 
backup teachers in treatment classrooms. In the case 
of one school, a research contact had to take over a 
classroom while serving in the role of research contact 
because a replacement teacher was never found. 

Studies also used incentives to decrease student 
attrition after scheduled school breaks (i.e., winter, 
spring break) and to reward students after successful 
assessment periods. The MPR study arranged pizza 
parties after breaks and testing to encourage 
student attendance, and the SFA program had built- 
in opportunities connected to student progress and 
attendance to earn points that could be spent to buy 
prizes. The incentives effectively encouraged student 
compliance with scheduled study activities. 

General Discussion 

Differences between grantee structure types— 
contractors, university policy center, and university- 
affiliated developer— played a role in the strengths 
and challenges the awardees brought to overcoming 
the problems associated with implementing rigorous 
research designs in afterschool settings. MPR, a 
private organization familiar with contract research 
and evaluations of programs, had benefited from first- 
year "pilot" data from which to implement the first 
year of the SEDL study. MPR's project director had 
built trust on the developer side as well as on the 
district side of the projects, with deep knowledge of 
the program and "real life" application of it in the 
school settings. This arrangement between evaluator 
and program represents a strong investment on all 
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sides of the project and a willingness to brainstorm 
and creatively solve problems. All these qualities came 
into play in the overall success of the project. 

The establishment of good communication patterns 
and trust between developers and evaluators also 
assisted the successful implementation of the 
programs. A critical set of conditions is essential, 
such as working from a common understanding and 
framework established between researchers, developers, 
as well as schools, regarding collection of assessment 
data, when it will be shared, what decisions will 
be made regarding implementation of the program 
based on fidelity data, and when the decisions will 
be implemented. The recruitment phase was a critical 
predictor of implementation success, influencing 
things like program administrators' understanding of 
study requirements critical to successful randomization 
fidelity and preservation of the sample. 

Conducting rigorous studies in applied afterschool 
settings requires that researchers balance the need to 
be flexible, while meeting critical benchmarks related 
to high quality research. Although the challenges 
encountered during the recruitment period ultimately 
delayed the start-up of the SFA study, recruitment 
efforts provided critical information about implementing 
high-quality ELL afterschool programs and also led 
to the enlistment of an administrator who may have 
been the key to the overall success of the recruitment 
phase. In the discussion of implementation, information 
is offered to uncover how the background conditions 
of the studies mattered for their implementation. 
Exploring the "black box" containing information 
about the implementation of these studies helps 
evaluation researchers identify critical areas for program 
improvement, such as the identification of "tolerable 



adaptations" (Cordray, 2008) to the intervention models 
in "real life" application. 

Unanticipated resource shortfalls for afterschool 
programs contributed to the loss of recruited sites in 
two of the three studies. The overall sample size for 
the CEEP study was significantly lower than initially 
projected for a number of valid reasons. Evaluators 
must, however, be able to reach and retain sample 
size targets that account for fairly high attrition rates 
seen in afterschool programs and meet the required 
size estimates for statistical power. Once too many 
sites are lost, or student attrition reaches dangerous 
levels, not much can be done to save the integrity of 
the design in RCT studies. The compressed timeframe 
between the start of the school year, the launch of 
afterschool program sites, and the release of project 
funds can hamper the project staffs ability to 
complete the required background research on program 
suitability. Although the RCT project staff engaged in a 
variety of processes to gain this knowledge before the 
implementation of the studies, there were important 
pieces of information that were gained along the way 
that were acted upon quickly and that ultimately 
contributed to successful project outcomes. 
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